Image collection and locating method, and image collection and locating device

ABSTRACT

Embodiments of the present application provide an image capturing-based positioning method and an image capturing-based positioning apparatus. The method comprises: determining that an eye of a user is gazing at an auxiliary positioning object; photographing the eye of the user and acquiring a distance of the user relative to the auxiliary positioning object; capturing at least one image comprising the auxiliary positioning object; obtaining, according to the at least one image, direction information of the user relative to the auxiliary positioning object and position information of the auxiliary positioning object; and obtaining position information of the user according to the position information of the auxiliary positioning object, the distance of the user relative to the auxiliary positioning object, and the direction information. In the embodiments of the present application, precise positioning is performed by acquiring a distance and a relative direction between a user and an auxiliary positioning object which the user is gazing at, to obtain a position of the user relative to the auxiliary positioning object, thereby improving the precision of image capturing-based positioning.

This application claims priority to Chinese Patent Application No.201310470129.4, filed with the Chinese Patent Office on Oct. 10, 2013and entitled “IMAGE COLLECTION AND LOCATING METHOD, AND IMAGE COLLECTIONAND LOCATING DEVICE”, which is incorporated herein by reference in itsentirety.

TECHNICAL FIELD

The present application relates to the field of positioningtechnologies, and in particular, to an image capturing-based positioningmethod and apparatus.

BACKGROUND

Positioning technologies have been widely applied, for example, theGlobal Positioning System (GPS) technology has been very mature, and iswidely applied in fields such as navigation. However, the GPS hasrelatively low positioning precision; moreover, the GPS has weak signalstrength in an indoor environment, and is not suitable for indoorapplications. Therefore, in recent years, many positioning methods,apparatuses and systems that are suitable for indoor environments anddifferent from the GPS are developed. For example, positioning isimplemented by the fading and the transmission delay characteristics ofa signal such as electromagnetic waves (Bluetooth, WIFI and the like),sound and visible light during transmission over the air. Positioningmay also be implemented according to different magnetic fielddistributions at different spatial locations. A direction and a distancefrom an object and a human body to a certain determined position may bededuced by tracking the movement of the object and the human with agyroscope and an accelerometer. Moreover, positioning may also beimplemented according to an image which is shot by a camera and has somefeature objects, where objects in the image have recognizable featuresand are located at specific positions, and therefore, the shot image maybe recognized first, and positioning may be performed by deducing theposition of the camera apparatus according to the image.

Mobile terminals having camera functions are currently very popular, andare easy to implement. Therefore, the positioning method based on animage shot by a camera has many applications, for example, US patentapplications NO. US20120176491 A1 and US20120209513 A2 both proposepositioning technologies based on an image shot by a camera.

However, the greatest problem of positioning based on an image shot by acamera is that: when the camera shoots an image, the position of thecamera may not be equivalent to the position of the shot image, andtherefore, after the position of the shot image is determined, relativepositions of the camera and the shot image need to be determined;otherwise, the positioning precision declines.

SUMMARY

One objective of the present application is to provide an imagecapturing-based positioning method and apparatus, so as to improveprecision of image capturing-based positioning.

In order to achieve the above objective, in a first aspect, the presentapplication provides an image capturing-based positioning method,comprising:

determining that an eye of a user is gazing at an auxiliary positioningobject;

photographing the eye of the user and acquiring a distance of the userrelative to the auxiliary positioning object;

capturing at least one image comprising the auxiliary positioningobject;

obtaining, according to the at least one image, direction information ofthe user relative to the auxiliary positioning object and positioninformation of the auxiliary positioning object; and

obtaining position information of the user according to the positioninformation of the auxiliary positioning object, the distance of theuser relative to the auxiliary positioning object, and the directioninformation.

In a second aspect, the present application further provides an imagecapturing-based positioning apparatus, comprising:

a gaze determining module, configured to determine that an eye of a useris gazing at an auxiliary positioning object;

an eye photographing module, configured to photograph the eye of theuser and acquire a distance of the user relative to the auxiliarypositioning object;

an image capturing module, configured to capture at least one imagecomprising the auxiliary positioning object;

an information acquisition module, configured to obtain, according tothe at least one image, direction information of the user relative tothe auxiliary positioning object, and position information of theauxiliary positioning object; and

a positioning module, configured to obtain position information of theuser according to the position information of the auxiliary positioningobject, the distance of the user relative to the auxiliary positioningobject, and the direction information.

In at least one technical solution embodiments of the presentapplication, precise positioning is performed by acquiring a distanceand a relative direction between a user and an auxiliary positioningobject which the user is gazing at, to obtain a position of the userrelative to the auxiliary positioning object, thereby improving theprecision of image capturing-based positioning.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of an image capturing-based positioning methodaccording to an embodiment of the present application;

FIG. 2a is a schematic diagram of a light spot pattern used in an imagecapturing-based positioning method according to an embodiment of thepresent application;

FIG. 2b is a schematic diagram of a fundus image having a light spotpattern and shot by using an image capturing-based positioning methodaccording to an embodiment of the present application;

FIG. 3 is a schematic structural block diagram of an imagecapturing-based positioning apparatus according to an embodiment of thepresent application;

FIG. 4 is a schematic structural block diagram of another imagecapturing-based positioning apparatus according to an embodiment of thepresent application;

FIG. 4a , FIG. 4b and FIG. 4c are schematic structural block diagrams ofthree information acquisition modules according to an embodiment of thepresent application;

FIG. 5a is a schematic structural block diagram of an eye photographingmodule according to an embodiment of the present application;

FIG. 5b is a schematic structural block diagram of another eyephotographing module according to an embodiment of the presentapplication;

FIG. 5c is a schematic diagram of an optical path for eye imaging of aneye photographing module according to an embodiment of the presentapplication;

FIG. 5d is a schematic diagram of obtaining, according to known imagingparameters of a system and optical parameters of an eye, a position ofan eye gaze point by an eye photographing module according to anembodiment of the present application;

FIG. 6 is a schematic diagram of applying an eye photographing module toa pair of glasses according to an embodiment of the present application;

FIG. 7 is a schematic diagram of applying another eye photographingmodule to a pair of glasses according to an embodiment of the presentapplication; and

FIG. 8 is a schematic structural diagram of an image capturing-basedpositioning apparatus according to an embodiment of the presentapplication.

DETAILED DESCRIPTION

The method and apparatus of the technical solutions of the presentapplication are described in detail below with reference to theaccompanying drawings and embodiments.

In a manner of performing positioning by using an image capturingmethod, some auxiliary positioning objects are selected in advance,where the auxiliary positioning objects may be ordinary objects (forexample, a vase) at fixed positions, and may also be some objects havingspecial identifiers (for example, a two-dimension code image); positionsof the auxiliary positioning objects are marked and stored, and when auser takes a photo of an auxiliary positioning object of which positioninformation is known, the auxiliary positioning object in the image isrecognized, and the position information of the auxiliary positioningobject is acquired for positioning of the user. However, the position ofthe auxiliary positioning object generally is not the position where theuser takes the photo, and therefore, the positioning method of directlyusing the position of the auxiliary positioning object as the positionof the user has low precision. Therefore, it is necessary to performprecise positioning to obtain a position of a user relative to anauxiliary positioning object, thereby obtaining a precise position ofthe user. As shown in FIG. 1, an embodiment of the present applicationprovides an image capturing-based positioning method, comprising:

S110: Determine that an eye of a user is gazing at an auxiliarypositioning object.

S120: Photograph the eye of the user, and acquire a distance of the userrelative to the auxiliary positioning object.

S130: Capture at least one image comprising the auxiliary positioningobject.

S140: Obtain, according to the at least one image, direction informationof the user relative to the auxiliary positioning object and positioninformation of the auxiliary positioning object.

S150: Obtain position information of the user according to the positioninformation of the auxiliary positioning object, the distance of theuser relative to the auxiliary positioning object, and the directioninformation.

In the embodiments of the present application, on one hand, bydetermining that the eye of the user is gazing at the auxiliarypositioning object, the distance of the user relative to the auxiliarypositioning object can be acquired precisely according to the eye gazepoint; on the other hand, the direction of the user relative to theauxiliary positioning object and the position information of theauxiliary positioning object are acquired by using the at least oneimage comprising the auxiliary positioning object; and finally, theposition information of the user is calculated according to the dataacquired. According to the method in the embodiments of the presentapplication, precise positioning may be performed to obtain the positionof the user relative to the auxiliary positioning object, therebyimproving the precision of positioning based on image capturing.

In a possible implementation of the embodiment of the presentapplication, in the S110, various methods may be used to determinewhether the user is gazing at the auxiliary positioning object, forexample, determining whether the eye is in a gaze state according tochanges of the eye and geometric parameters at the center of an eyeball,or determining whether the eye is in a gaze state based on features ofan image formed at the fundus (the two determining manners belong to theprior art). Then, it is determined, according to a sight line directionof the user, whether the user is gazing at the auxiliary positioningobject. For example, it may be determined whether the user is gazing atthe auxiliary positioning object by using a method described in thearticle “Study on SVM-Based Detection for Gaze of a Human Eye” publishedin the Journal of Optoelectronics Laser, Vol. 15 No. 10, in October,2004.

In order to help the user to notice the auxiliary positioning object andgaze at it, in a possible implementation of the embodiment of thepresent application, the method further comprises:

guiding the user to gaze at the auxiliary positioning object.

For example, in a possible implementation, the auxiliary positioningobject is marked to guide the user to gaze at the auxiliary positioningobject. For example, by using text or a special symbol on an object, theuser is reminded that the object is an auxiliary positioning object; foranother example, by setting a signal sending apparatus on an auxiliarypositioning object, a signal sent by the signal sending apparatus can bereceived near the auxiliary positioning object, which can be used toguide the user to gaze at the auxiliary positioning object.

In addition, in a possible implementation, the auxiliary positioningobject may be marked by means of augmented reality, to guide the user togaze at the auxiliary positioning object. For example, a pair of smartglasses may be used to provide some prompts related to the auxiliarypositioning object (for example, a two-dimensional code adhered at afixed position, or an object placed at a fixed position) in the field ofview by means of augmented reality, to guide a focal point of a humaneye to fall on the auxiliary positioning object.

In a possible implementation of an embodiment of the presentapplication, in the S120, various methods can be used to photograph theeye of the user and acquire the distance of the user relative to theauxiliary positioning object, comprising, for example, one of thefollowing methods:

i) A pupil direction detector is used to detect an optical axisdirection of an eye, then a depth sensor (for example, infrared distancemeasurement) is used to obtain depth information of each object in thefield of view of the eye, and an object at which the user is gazing atin the field of view can be determined. This technology belongs to theprior art, and is not repeated in this implementation. In theembodiments of the present application, the distance of the userrelative to the auxiliary positioning object may be obtained accordingto the depth information of the object in the field of view.

ii) Optical axis directions of two eyes are separately detected, thensight line directions of the two eyes of the user are obtained accordingto the optical axis directions of the two eyes, and a position of a gazepoint of sight lines of the eyes relative to the user is obtainedaccording to an intersection of the sight line directions of the twoeyes. This technology also belongs to the prior art, and is notdescribed in detail herein. In the embodiments of the presentapplication, according to the position of the gaze point of sight linesof the eyes relative to the user, the distance of the user relative tothe auxiliary positioning object can be obtained by means of geometriccalculation.

iii) According to optical parameters of an optical path between a fundusimage capturing sub-module and an eye and optical parameters of the eyewhen a fundus image captured corresponding to an image presented on animaging surface of the eye satisfying at least one set resolutioncriterion, a distance of a gaze point of a sight line of the eyerelative to the user is obtained. In the embodiments of the presentapplication, detailed procedures of the method are described in thefollowing, and are not described in detail herein.

Certainly, persons skilled in the art may know that, in addition to themethods in the foregoing forms, other methods for detecting a distanceof a gaze point of an eye of a user relative to the user may also beapplied to the method in the embodiments of the present application.

Detecting a current gaze point position of the user by using the methodiii) comprises:

S121: Capture at least one fundus image of the eye of the user.

S122: Adjust at least one imaging parameter of an optical path between acapturing position of the at least one fundus image and the eye until afundus image satisfying at least one set resolution criterion iscaptured.

S123: Analyze the least one fundus image to obtain imaging parameters ofthe optical path and optical parameters of the eye corresponding to thefundus image, and acquiring a distance of a current gaze point of theuser relative to the user according to the imaging parameters and theoptical parameters of the eye.

The resolution criterion described herein may be set according toresolution measurement parameters commonly used by persons skilled inthe art, for example, parameters such as effective resolution of animage, which is not described in detail herein.

In this embodiment, the at least one fundus image of the eye is analyzedand processed, to obtain the optical parameters of the eye when anfundus image satisfying at least one set resolution criterion iscaptured, and the imaging parameters of the corresponding optical pathare acquired as well, thereby obtaining the distance of the currentfocus point of the sight line relative to the user by means ofcalculation.

The image presented at the “fundus” is mainly an image presented on theretina, which may be an image of the fundus, or may be an image ofanother object projected onto the fundus, for example, a light spotpattern mentioned in the following.

In the S122, by adjusting the focal length of at least one opticaldevice on the optical path and/or the position of the at least oneoptical device on the optical path, the fundus image satisfying at leastone set resolution criterion can be acquired when the optical device isat a certain position or in a certain state. The adjustment may becontinuous real-time adjustment.

In a possible implementation of the method in the embodiments of thepresent application, the optical device may be a focal-length adjustablelens, configured to adjust the focal length thereof by adjusting therefractive index and/or shape of the optical device. Specifically: 1)the focal length is adjusted by adjusting the curvature of at least oneside of the focal-length adjustable lens, for example, the curvature ofthe focal-length adjustable lens is adjusted by adding or reducingliquid medium in a cavity formed by two transparent layers; and 2) thefocal length is adjusted by changing the refractive index of thefocal-length adjustable lens, for example, a specific liquid crystalmedium is filled in the focal-length adjustable lens, and arrangement ofthe liquid crystal medium is adjusted by adjusting a voltage of acorresponding electrode of the liquid crystal medium, thereby changingthe refractive index of the focal-length adjustable lens.

In another possible implementation of the method in one embodiment ofthe present application, the optical device may be: a lens assembly,configured to adjust relative positions between lenses in the lensassembly so as to adjust the focal length of the lens assembly.Alternatively, one or more lenses in the lens assembly are thefocal-length adjustable lenses described above.

In addition to changing the imaging parameters of the system by changingcharacteristics of the optical device as described above, the imagingparameters of the system may also be changed by adjusting the positionof the optical device on the optical path.

In addition, in the method of the embodiment of the present application,the S123 further comprises:

S1231: Analyze the at least one fundus image, to find the fundus imagesatisfying at least one set resolution criterion.

S1232: Calculate optical parameters of the eye according to the fundusimage, and imaging parameters of the optical path already known when thefundus image satisfying at least one set resolution criterion isobtained.

The adjustment in the S122 ensures that a fundus image satisfying atleast one set resolution criterion can be captured, but the S123 isneeded to find the fundus image satisfying at least one set resolutioncriterion in the at least one fundus image, and the optical parametersof the eye can be calculated according to the fundus image satisfying atleast one set resolution criterion and the known imaging parameters ofthe optical path.

In the method of the embodiment of the present application, the S123 mayfurther comprise:

S1233: Project a light spot to the fundus. The projected light spot mayhave no specific patterns but is only used for lightening the fundus.The projected light spot may also be a light spot pattern with abundantfeatures. The pattern with abundant features may be conducive todetection, and improve the detection precision. FIG. 2a is an exemplarydiagram of a light spot pattern 200, where the pattern may be formed bya light spot pattern generator, for example, frosted glass; and FIG. 2bshows a fundus image captured when the light spot pattern 200 isprojected.

To avoid affecting normal viewing of the eye, the light spot is aninfrared light spot invisible to the eye. Moreover, in order to reduceinterference of other spectrums, a step of filtering out light, exceptlight that can transmit through an eye-invisible light transmissionfilter, in the projected light spot may be performed.

Correspondingly, the method of the embodiment of the present applicationmay further comprise the following:

S1234: Control the brightness of the projected light spot according to aresult of the analysis of the S1231. The result of the analysiscomprises, for example, characteristics of the image captured in theS121, including the contrast of image features, texture features, andthe like.

It should be noted that, a special situation of controlling thebrightness of the projected light spot is starting or stopping theprojection, for example, when the user gazes at a point continuously,the projection may be stopped periodically; when the fundus of the useris bright enough, the projection may be stopped, and the distance fromthe current focus point of the sight line of the eye to the eye isdetected by using fundus information.

In addition, the brightness of the projected light spot may becontrolled according to ambient light.

In the method of the embodiment of the present application, the S123comprises:

S1235: Calibrate the fundus image to obtain at least one reference imagecorresponding to the image presented at the fundus. Specifically,comparison calculation is performed on the at least one image and thereference image, so as to obtain the fundus image satisfying at leastone set resolution criterion. Here, the fundus image satisfying at leastone set resolution criterion may be an obtained image having a minimumdifference with the reference image. In the method of thisimplementation, a difference between the currently acquired image andthe reference image is calculated by using an existing image processingalgorithm, for example, using a classical phase difference automaticfocusing algorithm.

In one embodiment of the present application, the optical parameters ofthe eye obtained in the S1232 may comprise an optical axis direction ofthe eye (in the following descriptions of the embodiment of the presentapplication, the optical axis direction of the eye is a direction of theoptical axis of the eye relative to a certain reference plane of theuser, for example, a direction relative to the front of the user)obtained according to the features of the eye when the fundus imagesatisfying at least one set resolution criterion is captured. Theoptical axis direction may be used in adjustment of an image capturingdirection in the S121 as described in the following, and definitely, ifit is unnecessary to adjust the image capturing direction, the opticalaxis direction may not be needed to be acquired. Here, the features ofthe eye may be acquired from the fundus image satisfying at least oneset resolution criterion, or may be acquired in other manners. Theoptical axis direction of the eye is corresponding to a gaze directionof a sight line of the eye. Specifically, the optical axis direction ofthe eye is obtained according to features of the fundus when the fundusimage satisfying at least one set resolution criterion is obtained.Determining the optical axis direction of the eye according to thefeatures of the fundus may be more precise.

When a light spot pattern is projected to the fundus, the area of thelight spot pattern may be greater than that of a visible region of thefundus or smaller than that of the visible region of the fundus, where:

when the area of the light spot pattern is smaller than or equal to thatof the visible region of the fundus, a classical feature point matchingalgorithm (for example, Scale Invariant Feature Transform (SIFT)algorithm) may be used to determine the optical axis direction of theeye by detecting a position of the light spot pattern on the imagerelative to the fundus.

When the area of the light spot pattern is greater than that of thevisible region of the fundus, the optical axis direction of the eye maybe determined according to a position of the light spot pattern on theimage relative to an original light spot pattern (acquired by means ofimage calibration), so as to determine a sight line direction of anobserver.

In another possible implementation of the method in the embodiments ofthe present application, the optical axis direction of the eye may alsobe obtained according to features of the pupil when the fundus imagesatisfying at least one set resolution criterion is obtained. Here, thefeatures of the pupil may be acquired from the fundus image satisfyingat least one set resolution criterion, and may also be acquired in othermanners. Obtaining the optical axis direction of the eye according tothe features of the pupil belongs to the prior art, and is not describedin detail herein.

Moreover, the method in the embodiments of the present application mayfurther comprise a step of calibrating the optical axis direction of theeye, so as to determine the optical axis direction of the eye moreprecisely.

In the method of the embodiments of the present application, the imagingparameters of the optical path between the eye and the capturingposition of the at least one fundus image can comprise at least onefixed imaging parameter and at least one real-time imaging parameter,where the at least one real-time imaging parameter is parameterinformation about the optical device when the fundus image satisfying atleast one set resolution criterion is acquired, and the parameterinformation may be obtained by means of real-time recording when thefundus image satisfying at least one set resolution criterion isacquired.

After the current optical parameters of the eye are obtained, thedistance from the eye focus point to the eye of the user may becalculated (specific procedures will be described in detail withreference to the apparatus part).

In a possible implementation of one embodiment of the presentapplication, in the S130 of capturing the image comprising the auxiliarypositioning object, the image capturing direction may directly face theauxiliary positioning object, or the capturing may be performed at acertain inclination angle.

To make sure that the image captured by the user comprises the auxiliarypositioning object, and to facilitate recognition of the auxiliarypositioning object in the image, in a possible implementation of theembodiment of the present application, the image capturing direction maybe adjusted according to the optical axis direction of the eye describedin the foregoing. For example, in some implementations, a direction ofthe sight line of the user relative to the user (briefly referred to asa sight line direction) is obtained according to the optical axisdirection of the eye, and the image capturing direction is adjusted tobe consistent with the sight line direction of the user; in this way,the image is captured with an object gazed at by the eye of the user asa center, so that subsequent recognition of the auxiliary positioningobject in the image is more convenient.

In the embodiments of the present application, in the S140, there arevarious methods for acquiring the direction information of the userrelative to the auxiliary positioning object and the positioninformation of the auxiliary positioning object according to the atleast one image. For example:

In a possible implementation, the S140 comprises:

sending the captured image to an external device; and

receiving, the direction information of the user relative to theauxiliary positioning object and/or the position information of theauxiliary positioning object.

For example, the direction information of the user relative to theauxiliary positioning object, or the position information of theauxiliary positioning object, or the direction information of the userrelative to the auxiliary positioning object and the positioninformation of the auxiliary positioning object may be acquired by usingan external positioning server.

The positioning server stores at least one image shot in a determineddirection and comprising the auxiliary positioning object, or storesposition information of the auxiliary positioning object, or stores boththe image comprising the auxiliary positioning object and the positioninformation of the auxiliary positioning object.

Specifically, in the method of the embodiment of the presentapplication, the at least one image is sent to the external positioningserver, and after the positioning server receives the at least oneimage, the positioning server searches for an pre-stored imagesatisfying a set matching degree to the at least one image in an imagelibrary (the image library comprises, for example, an pre-stored imagecorresponding to the auxiliary positioning object in the at least oneimage, and pre-stored images corresponding to other auxiliarypositioning objects).

When the position information of the auxiliary positioning object needsto be acquired, the positioning server may acquire position informationcorresponding to the auxiliary positioning object in the at least oneimage according to the image satisfying the set matching degree, and thepositioning server then returns the position information to a user side,for example, sends the position information to an intelligent devicecarried by the user.

When the direction information of the user relative to the auxiliarypositioning object needs to be acquired:

In a possible implementation, the positioning server performs rotationanalysis on a most matched image in at least one pre-stored image storedin the image library according to the at least one image, until the mostmatched image has a highest matching degree with the at least one image.According to a photographing angle and a rotation angle of the mostmatched image, the direction of the capturing position of the at leastone image relative to the auxiliary positioning object can be obtained.The capturing position of the at least one image is generally near theuser (for example, the user captures an image by using a portable devicesuch as a camera, a pair of smart glasses, and a mobile phone), andtherefore, the capturing position of the at least one image can beconsidered as the position where the user is located; or, to make thepositioning more precise, the position of the user may be acquiredaccording to the capturing position of the at least one image.

In another possible implementation, the at least one pre-stored imageand the at least one image each comprise multiple auxiliary positioningobjects (or comprise one auxiliary positioning object and otherobjects), and the image capturing direction may be obtained according tochanges of positions of the multiple auxiliary positioning objects inthe pre-stored image and the at least one image, thereby acquiring thedirection information of the user relative to the auxiliary positioningobject.

In still another possible implementation, the at least one pre-storedimage comprise a panorama of the current scenario (for example, apanorama of a museum), and analysis is performed according to thepanorama and the at least one image, so as to obtain the image capturingdirection (this method belongs to the prior art, and is not described indetail herein).

Certainly, persons skilled in the art may know that, in addition to theabove methods, the direction information of the user relative to theauxiliary positioning object may also be acquired by using other imageanalysis methods.

In addition to acquiring the direction information of the user relativeto the auxiliary positioning object and/or the position information ofthe auxiliary positioning object by using the external positioningserver as described above, the direction information of the userrelative to the auxiliary positioning object and/or the positioninformation of the auxiliary positioning object may also be obtained bylocally pre-storing the at least one pre-stored image and analyzing theat least one image. Specifically, in a possible implementation, the S140comprises:

analyzing the at least one image to obtain the direction information ofthe user relative to the auxiliary positioning object.

The method comprises: pre-storing at least one pre-stored image shot inat least one determined direction and comprising the auxiliarypositioning object.

In the S140, the at least one image is analyzed according to the atleast one pre-stored image, so as to obtain the direction information ofthe user relative to the auxiliary positioning object.

In a possible implementation, the S140 comprises:

recognizing the auxiliary positioning object in the at least one image;and

acquiring position information of the auxiliary positioning object whichis pre-stored.

The image analysis performed locally is basically similar to the imageanalysis performed at the side of the positioning server, and is notdescribed in detail herein.

After the position information of the auxiliary positioning object, thedistance of the user relative to the auxiliary positioning object, andthe direction information of the user relative to the auxiliarypositioning object are acquired by performing the S120 and the S130, theembodiment of the present application further comprises the S150, inwhich position information of the user may be obtained by means ofmathematic calculation according to the position information of theauxiliary positioning object, the distance of the user relative to theauxiliary positioning object, and the direction information.

In some possible implementations of embodiments of the presentapplication, considering that the precision of a positioning result isaffected when an included angle is formed between the sight linedirection of the eye and a horizontal plane, in the method of thepresent application, the positioning result may further be modified by acertain extent after the included angle is determined by using, forexample, a three-dimensional direction sensor or an image processingmethod, so that the positioning result is more precise.

It should be understood that, in various embodiments of the presentapplication, the serial numbers of the steps do not represent thesequence of performing the steps, and the sequence of performing theprocesses should be determined by functions and internal logic thereof,and should not constitute any limit to the implementation process of theembodiment of the present application.

As shown in FIG. 3, an embodiment of the present application furtherprovides an image capturing-based positioning apparatus 300, comprising:

a gaze determining module 310, configured to determine that an eye of auser is gazing at an auxiliary positioning object;

an eye photographing module 320, configured to photograph the eye of theuser and acquire a distance of the user relative to the auxiliarypositioning object;

an image capturing module 330, configured to capture at least one imagecomprising the auxiliary positioning object;

an information acquisition module 340, configured to obtain, accordingto the at least one image, direction information of the user relative tothe auxiliary positioning object, and position information of theauxiliary positioning object; and

a positioning module 350, configured to obtain position information ofthe user according to the position information of the auxiliarypositioning object, the distance of the user relative to the auxiliarypositioning object, and the direction information.

In the embodiments of the present application, on one hand, the gazedetermining module 310 determines that the eye of the user is gazing atthe auxiliary positioning object, and the distance of the user relativeto the auxiliary positioning object may be acquired precisely accordingto the eye gaze point; on the other hand, the direction of the userrelative to the auxiliary positioning object and the positioninformation of the auxiliary positioning object are acquired by usingthe at least one image comprising the auxiliary positioning object; andfinally, the positioning module 350 calculates the position informationof the user according to the data acquired. According to the apparatusin the embodiment of the present application, precise positioning may beperformed to obtain the position of the user relative to the auxiliarypositioning object, thereby improving the precision of the imagecapturing-based positioning.

In the embodiments of the present application, the gaze determiningmodule 310 may be any one of various structures for determining whetherthe user is gazing at the auxiliary positioning object, for example, astructure for determining whether the user is gazing at an objectaccording to changes of the eye and geometric parameters at the centerof an eyeball, or a structure for determining, based on features of animage formed at the fundus, whether the user is gazing at an object (thetwo structures belong to the prior art). Then, it is determined,according to a sight line direction of the user, whether the user isgazing at the auxiliary positioning object.

As shown in FIG. 4, in order to help a user to notice the auxiliarypositioning object and gaze at it, in a possible implementation of oneembodiment of the present application, the apparatus 300 furthercomprises:

an object guiding module 360, configured to guide the user to gaze atthe auxiliary positioning object.

In this embodiment, the function of the object guiding module 360 isimplemented according to the corresponding description in the methodembodiment shown in FIG. 1, and is not repeated in this embodiment.

In a possible implementation, the object guiding module 360 furthercomprises:

an object marking unit 361, configured to mark the auxiliary positioningobject.

In a possible implementation of one embodiment of the presentapplication, the apparatus 300 may be a portable or wearable devicehaving a mixed reality function, for example, a pair of smart glasses.In a possible implementation of the embodiment of the presentapplication, the object marking unit comprises:

an augmented reality marking sub-unit 3611, configured to mark theauxiliary positioning object by means of augmented reality.

For example, on an image comprising the auxiliary positioning object andshot in real time by a pair of smart glasses, the auxiliary positioningobject is marked by means of augmented reality such as highlighting, ordisplaying a symbol or text. The function of the unit is implementedaccording to the corresponding description in the method embodimentshown in FIG. 1, and is not described in detail herein.

In the embodiments of the present application, the manner in which theeye photographing module 320 photographs the eye of the user andacquires the distance of the user relative to the auxiliary positioningobject may be any one or more of the methods i) to iii) mentioned in themethod embodiment shown in FIG. 1. The apparatus in one embodiment ofthe present application adopts the manner iii) which has higherdetection precision, and correspondingly, the eye photographing modulemay be any one of eye photographing modules shown in FIG. 5a to FIG. 5d, FIG. 6, and FIG. 7.

Certainly, persons skilled in the art may know that, in addition to theabove forms of eye photographing modules, other apparatuses that can beused for photographing the eye of the user and acquiring the distance ofthe user relative to the auxiliary positioning object may also beapplied to the apparatus in the embodiments of the present application.

The eye photographing module of the form iii) is further described asfollows:

As shown in FIG. 5a , in a possible implementation of the embodiment ofthe present application, the eye photographing module 500 comprises:

a fundus image capturing sub-module 510, configured to capture at leastone fundus image of the eye;

an adjustable imaging sub-module 520, configured to adjust at least oneimaging parameter of an optical path between the fundus image capturingsub-module 510 and the eye, so that the fundus image capturingsub-module 510 can capture a fundus image satisfying at least one setresolution criterion; and

an image processing sub-module 530, configured to analyze the at leastone fundus image, to obtain imaging parameters of the optical path andoptical parameters of the eye corresponding to the fundus image, andcalculate a distance of an eye gaze point relative to the user accordingto the imaging parameters and the optical parameters of the eye.

In this implementation, the eye photographing module 500 analyzes the atleast one fundus image of the eye to obtain the optical parameters ofthe eye when the fundus image capturing sub-module captures the fundusimage satisfying at least one set resolution criterion, and therefore,can calculate the distance of the current eye gaze point relative to theuser.

The image presented at the “fundus” is mainly an image presented on theretina, which may be an image of the fundus, or may be an image ofanother object projected to the fundus. Here, the eye may be a humaneye, and may also be an eye of another animal.

As shown in FIG. 5b , in a possible implementation of one embodiment ofthe present application, the fundus image capturing sub-module 510 is amicro camera, and in another possible implementation of one embodimentof the present application, a photosensitive imaging device, such as aCCD or a CMOS, may also be directly used as the fundus image capturingsub-module 510.

In a possible implementation of the embodiments of the presentapplication, the adjustable imaging sub-module 520 comprises: anadjustable lens device 521, located on the optical path between the eyeand the fundus image capturing sub-module 510, and having an adjustablefocal length and/or an adjustable position on the optical path. By usingthe adjustable lens device 521, a system equivalent focal length betweenthe eye and the fundus image capturing sub-module 510 may be adjusted,and by adjusting the adjustable lens device 521, the fundus imagecapturing sub-module 510 captures a fundus image satisfying at least oneset resolution criterion when the adjustable lens device 521 is at acertain position or in a certain state. In this implementation, theadjustable lens device 521 performs adjustment continuously in real timeduring detection.

In a possible implementation of the embodiments of the presentapplication, the adjustable lens device 521 is a focal-length adjustablelens, configured to adjust the focal length thereof by adjusting therefractive index and/or shape thereof. Specifically: 1) the focal lengthis adjusted by adjusting the curvature of at least one side of thefocal-length adjustable lens, for example, the curvature of thefocal-length adjustable lens is adjusted by adding or reducing liquidmedium in a cavity formed by two transparent layers; and 2) the focallength is adjusted by changing the refractive index of the focal-lengthadjustable lens, for example, a specific liquid crystal medium is filledin the focal-length adjustable lens, and arrangement of the liquidcrystal medium is adjusted by adjusting a voltage of a correspondingelectrode of the liquid crystal medium, thereby changing the refractiveindex of the focal-length adjustable lens.

In another possible implementation of the embodiment of the presentapplication, the adjustable lens device 521 comprises: a lens assemblyformed by multiple lenses, configured to adjust relative positionsbetween lenses in the lens assembly so as to adjust the focal length ofthe lens assembly. The lens assembly may also comprise a lens havingadjustable imaging parameters such as the focal length thereof.

In addition to changing optical path parameters of the eye photographingmodule by adjusting characteristics of the adjustable lens device 521 asdescribed in the foregoing, the optical path parameters of the eyephotographing module may also be changed by adjusting a position of theadjustable lens device 521 on the optical path.

In a possible implementation of the embodiments of the presentapplication, to avoid affecting experience of the user viewing anobserved object, and to enable the eye photographing module to beportably applied to a wearable device, the adjustable imaging sub-module520 may also comprise: a beam splitting unit 522, configured to formlight transmission paths between the eye and the observed object andbetween the eye and the fundus image capturing sub-module 510.Therefore, the optical path can be folded to reduce the volume of theeye photographing module and avoid affecting other visual experience ofthe user as far as possible.

In this implementation, the beam splitting unit 522 comprises: a firstbeam splitting unit, located between the eye and the observed object,and configured to transmit light from the observed object to the eye andtransmit light from the eye to the fundus image capturing sub-module510.

The first beam splitting unit may be a beam splitter, a beam splittingoptical waveguide (including an optical fiber) or another suitable beamsplitting device.

In a possible implementation of the embodiments of the presentapplication, the image processing sub-module 530 of the eyephotographing module comprises an optical path calibration unit,configured to calibrate the optical path of the eye photographingmodule, for example, align an optical axis of the optical path, toensure the precision of the measurement.

In a possible implementation of the embodiments of the presentapplication, the image processing sub-module 530 comprises:

an image analyzing unit 531, configured to analyze the at least onefundus image, to find the fundus image satisfying at least one setresolution criterion; and

a parameter calculating unit 532, configured to calculate opticalparameters of the eye according to the fundus image and the knownimaging parameters of the system when the fundus image satisfying atleast one set resolution criterion is obtained.

In this implementation, by using the adjustable imaging sub-module 520,the fundus image capturing sub-module 510 can obtain the fundus imagesatisfying at least one set resolution criterion; however, the fundusimage satisfying at least one set resolution criterion in the at leastone fundus image needs to be found by using the image analyzing unit531, and at this time, the optical parameters of the eye can becalculated according to the fundus image satisfying at least one setresolution criterion and the known optical parameters of the system. Theoptical parameters of the eye may comprise an optical axis direction ofthe eye.

In a possible implementation of the embodiments of the presentapplication, the eye photographing module further comprises: aprojection sub-module 540, configured to project a light spot to thefundus. In a possible implementation, the function of the projectionsub-module may be implemented by using a mini projector.

The projected light spot may have no specific patterns but is only usedfor lightening the fundus.

In a preferred implementation of the embodiments of the presentapplication, the projected light spot comprises a pattern with abundantfeatures. The pattern with abundant features may be conducive todetection, and improve the detection precision. FIG. 2a is an exemplarydiagram of a light spot pattern 200, where the pattern may be formed bya light spot pattern generator, for example, frosted glass; and FIG. 2bshows a fundus image shot when the light spot pattern 200 is projected.

To avoid affecting normal viewing of the eye, the light spot is aninfrared light spot invisible to the eye.

Moreover, in order to reduce interference of other spectrums:

an emergent surface of the projection sub-module may be provided with aneye-invisible light transmission filter; and

an incident surface of the fundus image capturing sub-module is providedwith an eye-invisible light transmission filter.

In a possible implementation of the embodiments of the presentapplication, the image processing sub-module 530 further comprises:

a projection control unit 534, configured to control, according to aresult obtained by the image analyzing unit, the brightness of the lightspot projected by the projection sub-module.

For example, the projection control unit 534 may self-adaptively adjustthe brightness according to characteristics of the at least one fundusimage captured by the fundus image capturing sub-module 510. Here, thecharacteristics of the image include the contrast of image features,texture features, and the like.

Here, a special situation of controlling the brightness of the lightspot projected by the projection sub-module is turning on or turning offthe projection sub-module, for example, when the user gazes at a pointcontinuously, the projection sub-module may be turned off periodically;and when the fundus of the user is bright enough, a light-emittingsource may be turned off, and the distance from the current eye gazepoint to the eye is detected only using fundus information.

In addition, the projection control unit 534 may further control thebrightness of the light spot projected by the projection sub-moduleaccording to ambient light.

In a possible implementation of the embodiments of the presentapplication, the image processing sub-module 530 further comprises: animage calibration unit 533, configured to calibrate a fundus image, toobtain at least one reference image corresponding to the image presentedat the fundus.

The image analyzing unit 531 performs comparison calculation on the atleast one image captured by the fundus image capturing sub-module 530and the reference image, to obtain the fundus image satisfying at leastone set resolution criterion. Here, the fundus image satisfying at leastone set resolution criterion may be an obtained image having a minimumdifference with the reference image. In this implementation, adifference between the currently acquired image and the reference imageis calculated by using an existing image processing algorithm, forexample, using a classical phase difference automatic focusingalgorithm.

In a possible implementation of the embodiments of the presentapplication, the parameter calculating unit 532 comprises:

an eye optical axis direction determining sub-unit 5321, configured toobtain an optical axis direction of the eye according to features of theeye when the fundus image satisfying at least one set resolutioncriterion is acquired.

The features of the eye may be acquired from the fundus image satisfyingat least one set resolution criterion, or may be acquired in othermanners. The optical axis direction of the eye is corresponding to agaze direction of a sight line of the eye.

In a possible implementation of the embodiments of the presentapplication, the eye optical axis direction determining sub-unit 5321comprises: a first determining sub-unit, configured to obtain theoptical axis direction of the eye according to features of the funduswhen the fundus image satisfying at least one set resolution criterionis obtained. As compared with obtaining the optical axis direction ofthe eye according to features of the pupil and eyeball surface,determining the optical axis direction of the eye according to thefeatures of the fundus is more precise.

When a light spot pattern is projected to the fundus, the area of thelight spot pattern may be greater than that of a visible region of thefundus or smaller than that of the visible region of the fundus, and forthe method for acquiring the optical axis direction of the eye in thetwo situations, reference may be made to the corresponding descriptionin the method embodiment shown in FIG. 1.

In another possible implementation of the embodiments of the presentapplication, the eye optical axis direction determining sub-unit 5321comprises: a second determining sub-unit, configured to obtain theoptical axis direction of the eye according to features of the pupilwhen the fundus image satisfying at least one set resolution criterionis obtained. Here, the features of the pupil may be acquired from thefundus image satisfying at least one set resolution criterion, and mayalso be acquired in other manners. Obtaining the optical axis directionof the eye according to the features of the pupil belongs to the priorart, and is not described in detail herein.

In a possible implementation of the embodiments of the presentapplication, the image processing sub-module 530 further comprises: aneye optical axis direction calibration unit 535, configured to calibratethe optical axis direction of the eye, to determine the optical axisdirection of the eye more precisely.

In the image capturing-based positioning apparatus according to thepresent application, the optical axis direction may be used by the imagecapturing module when adjusting the image capturing direction, anddefinitely, if the image capturing direction does not need to beadjusted, the optical axis direction may not be acquired, and the fundusimage photographing apparatus 500 may not be provided with the eyeoptical axis direction determining sub-unit or the eye optical axisdirection calibration unit.

In the method of the embodiments of the present application, the imagingparameters of the optical path between the eye and the capturingposition of the at least one fundus image comprise at least one fixedimaging parameter and at least one real-time imaging parameter, wherethe at least one real-time imaging parameter is parameter informationabout the optical device when the fundus image satisfying at least oneset resolution criterion is acquired, and the parameter information maybe obtained by means of real-time recording when the fundus imagesatisfying at least one set resolution criterion is acquired.

After the current optical parameters of the eye are obtained, thedistance from the eye gaze point to the user may be calculated, andspecifically:

FIG. 5c shows a schematic diagram of eye imaging, and in combinationwith a lens imaging formula in the classical optical theory, formula (1)can be obtained from FIG. 5c :

$\begin{matrix}{{\frac{1}{d_{o}} + \frac{1}{d_{e}}} = \frac{1}{f_{e}}} & (1)\end{matrix}$

where d_(o) and d_(e) are respectively a distance from a currentobserved object 5010 of the eye to an eye equivalent lens 5030 and adistance from a real image 5020 on the retina to the eye equivalent lens5030, f_(e) is an equivalent focal length of the eye equivalent lens5030, and X is a sight line direction of the eye (which may be obtainedaccording to the optical axis direction of the eye).

FIG. 5d shows a schematic diagram of obtaining a distance from an eyegaze point to the eye according to known optical parameters of thesystem and optical parameters of the eye. In FIG. 5d , a light spot 5040forms a virtual image (not shown in FIG. 5d ) through the adjustablelens device 521; assuming that a distance between the virtual image andthe lens is x (not shown in FIG. 5d ), the following equation set may beobtained in combination with formula (1):

$\begin{matrix}\{ \begin{matrix}{{\frac{1}{d_{p}} - \frac{1}{x}} = \frac{1}{f_{p}}} \\{{\frac{1}{d_{i} + x} + \frac{1}{d_{e}}} = \frac{1}{f_{e}}}\end{matrix}  & (2)\end{matrix}$

where d_(p) is an optical equivalent distance from the light spot 5040to the adjustable lens device 521, d_(i) is an optical equivalentdistance from the adjustable lens device 521 to the eye equivalent lens5030, f_(p) is a focal length value of the adjustable lens device 521,and d_(i) is a distance from the eye equivalent lens 5030 to theadjustable lens device 521.

According to (1) and (2), a distance d_(o) from the current observedobject 5010 (eye gaze point) to the eye equivalent lens 5030 is as shownin formula (3):

$\begin{matrix}{d_{o} = {d_{i} + \frac{d_{p} \cdot f_{p}}{f_{p} - d_{p}}}} & (3)\end{matrix}$

According to the distance from the observed object 5010 to the eyecalculated above, and the optical axis direction of the eye obtainedaccording to the foregoing description, the position of the eye gazepoint may be obtained easily, providing a basis for subsequent furtherinteraction related to the eye.

FIG. 6 shows an embodiment of applying an eye photographing module 600according to a possible implementation of an embodiment of the presentapplication to a pair of glasses 400, which comprises content describedin the implementation shown in FIG. 5b , and specifically, it can beseen from FIG. 6 that, in this implementation, the apparatus 600 of thisimplementation is integrated to the right side (not limited thereto) ofthe glasses 400, and comprises:

a micro camera 610, which functions the same as the fundus imagecapturing sub-module described in the implementation of FIG. 5b , and isdisposed at the outer right side of the glasses 400 to avoid affectingthe sight line of normal viewing of the user;

a first beam splitter 620, which functions the same as the first beamsplitting unit described in the implementation of FIG. 5b , is disposedwith a certain inclination angle at an intersection of a gaze directionof an eye A and an incident direction of the camera 610, and transmitslight from an observed object to the eye A and reflects light from theeye to the camera 610; and

a focal-length adjustable lens 630, which functions the same as thefocal-length adjustable lens described in the implementation of FIG. 5b, is located between the first beam splitter 620 and the camera 610, andadjusts a focal length value in real time, so that the camera 610 canshoot a fundus image satisfying at least one set resolution criterion ata certain focal length value.

In this implementation, the image processing sub-module is not shown inFIG. 6, and functions the same as the image processing sub-module shownin FIG. 5 b.

Generally, the brightness at the fundus is insufficient, and therefore,the fundus is lightened preferably. In this implementation, alight-emitting source 640 is used to lighten the fundus. Thelight-emitting source 640 is preferably an invisible light-emittingsource, so as to avoid affecting the experience of the user, andpreferably, a near-infrared light-emitting source which has small impacton the eye A and to which the camera 610 is relatively sensitive isused.

In this implementation, the light-emitting source 640 is located at theouter side of a spectacle frame at the right side, and therefore, asecond beam splitter 650 together with the first beam splitter 620 isrequired to transmit light emitted by the light-emitting source 640 tothe fundus. In this implementation, the second beam splitter 650 islocated in front of the incident surface of the camera 610, andtherefore, the incident surface further needs to transmit the light fromthe fundus to the second beam splitter 650.

It can be seen that, in this implementation, in order to improve theuser experience and improve the capture definition of the camera 610,the first beam splitter 620 may preferably have characteristics of highreflectivity to infrared and high transmissivity to visible light. Forexample, an infrared reflective film may be disposed at one side of thefirst beam splitter 620 facing the eye A, so as to implement theforegoing characteristics.

It can be seen from FIG. 6 that, in this implementation, the eyephotographing module 600 is located at one side, away from the eye A, ofthe lens of the glasses 400, and therefore, when the optical parametersof the eye are calculated, the lens can be considered as a part of theeye A, and it is unnecessary to know optical characteristics of thelens.

In other implementations of the embodiments of the present application,the eye photographing module 600 may be located at one side, near theeye A, of the lens of the glasses 400, and at this time, it is requiredto obtain optical characteristic parameters of the lens, and influencingfactors of the lens are taken into consideration when a gaze pointdistance is calculated.

In this embodiment, the light emitted by the light-emitting source 640is reflected by the second beam splitter 650, projected by thefocal-length adjustable lens 630, reflected by the first beam splitter620, then transmits through the lens of the glasses 400 to enter intothe eye of the user, and finally reaches the retina at the fundus. Thecamera 610 shoots a fundus image through the pupil of the eye A along anoptical path formed by the first beam splitter 620, the focal-lengthadjustable lens 630 and the second beam splitter 650.

FIG. 7 is a schematic structural diagram of another implementation of aneye photographing module 700 according to an embodiment of the presentapplication. It can be seen from FIG. 7 that, this implementation issimilar to the implementation shown in FIG. 6, and comprises a microcamera 710, a second beam splitter 720, and a focal-length adjustablelens 730; this implementation differs from the implementation shown inFIG. 6 in that, a projection sub-module 740 in this implementation isthe projection sub-module 740 for projecting a light spot pattern, and acurved-surface beam splitter 750, used as a curved-surface beamsplitting device, replaces the first beam splitter in the implementationof FIG. 6.

The curved-surface beam splitter 750 separately corresponds to pupilpositions associated with different optical axis directions of the eye,and transmits an image presented at the fundus to the fundus imagecapturing sub-module. In this way, the camera can capture mixed andsuperimposed eyeball images formed at various angles. However, only thefundus part passing through the pupil can be imaged clearly on thecamera, and other parts are out of focus and cannot be imaged, and donot severely affect the imaging of the fundus part; therefore, featuresof the fundus part can still be detected. As a result, compared with theimplementation shown in FIG. 6, this implementation can obtain goodfundus images when the eye is gazing at different directions, so thatthe eye photographing module of this implementation has a broaderapplication range and higher detection precision.

In a possible implementation of the embodiments of the presentapplication, the image capturing module 330 may be, for example, acamera on a pair of smart glasses, or may also be a camera module of adevice carried by the user.

To make sure that the at least one image captured by the image capturingmodule 330 comprises the auxiliary positioning object, or to facilitaterecognition of the auxiliary positioning object in the image, in apossible implementation of the embodiments of the present application,the image capturing-based positioning apparatus may further comprise animage capturing direction adjusting module 370, configured to adjust theimage capturing direction according to the optical axis direction of theeye described above. For example, in some implementations, the directionof the sight line of the user relative to the user is obtained accordingto the optical axis direction of the eye, and the image capturingdirection is adjusted to be consistent with the sight line direction ofthe user; in this way, the image is captured with an object gazed at bythe eye of the user as a center, so that subsequent recognition of theauxiliary positioning object in the image is more convenient.

As shown in FIG. 4a , in a possible implementation of the embodiments ofthe present application, the information acquisition module 340comprises a communication unit 341, configured to:

send the at least one image to an external device; and

receive, from the external device, the direction information of the userrelative to the auxiliary positioning object and/or the positioninformation of the auxiliary positioning object.

For the method of analyzing the at least one image and acquiring thedirection information of the user relative to the auxiliary positioningobject and/or the position information of the auxiliary positioningobject by an external positioning server (used as the external device),reference may be made to the corresponding descriptions in the methodembodiment shown in FIG. 1, which is not described in detail herein.

As shown in FIG. 4b , in another possible implementation of theembodiments of the present application, the information acquisitionmodule 340 comprises:

an image analyzing unit 342, configured to analyze the at least oneimage, so as to obtain the direction information of the user relative tothe auxiliary positioning object.

In another possible implementation of the embodiments of the presentapplication, the information acquisition module 340 further comprises:

a storage unit 343, configured to pre-store at least one pre-storedimage shot in at least one determined direction and comprising theauxiliary positioning object.

The image analyzing unit 342 analyzes the at least one image accordingto the at least one pre-stored image, to obtain the directioninformation of the user relative to the auxiliary positioning object.

As shown in FIG. 4c , in another possible implementation of theembodiments of the present application, the information acquisitionmodule 340 comprises:

an object recognition unit 344, configured to recognize the auxiliarypositioning object in the at least one image; and

an object information acquisition unit 345, configured to acquireposition information of the auxiliary positioning object which ispre-stored.

The functions of the units of the information acquisition module 340 areimplemented according to the corresponding descriptions in the foregoingmethod embodiment, and are not described in detail herein.

In a possible implementation of the embodiments of the presentapplication, the apparatus is a pair of smart glasses. The camera of thesmart glasses is very close to the eye of the user, and it can beconsidered that an image captured by the camera is the image that shouldappear in the visual field of the user, and correction between thecapturing position of the at least one image and the user position isnot required, so that the user positioning is more natural and precise.

In addition, some embodiments of the present application may furtherprovide a computer readable medium, comprising computer readableinstructions for performing the following operations when beingexecuted: operations of the S110, S120, S130, S140 and S150 in themethod of the foregoing embodiment.

FIG. 8 is a schematic structural diagram of still another imagecapturing-based positioning apparatus 800 according to an embodiment ofthe present application, and specific implementation of the imagecapturing-based positioning apparatus 800 is not limited in the specificembodiment of the present application. As shown in FIG. 8, the imagecapturing-based positioning apparatus 800 may comprise:

a processor 810, a communications interface 820, a memory 830, and acommunication bus 840.

The processor 810, the communications interface 820 and the memory 830communicate with each other through the communication bus 840.

The communications interface 820 is configured to communicate with anetwork element such as a client.

The processor 810 is configured to execute a program 832, andspecifically, the processor may execute related steps in the methodembodiment shown in FIG. 1.

Specifically, the program 832 may comprise program code, and the programcode comprises computer operation instructions.

The processor 810 may be a central processing unit (CPU), or anapplication specific integrated circuit (ASIC), or is configured as oneor more integrated circuits for implementing the embodiment of thepresent application.

The memory 830 is configured to store the program 832. The memory 830may comprise a high-speed RAM memory, and may also comprise anon-volatile memory, for example, at least one magnetic disk memory.Specifically, the program 832 may enable the image capturing-basedpositioning apparatus to perform the following operations:

determining that an eye of a user is gazing at an auxiliary positioningobject;

capturing an image corresponding to the eye of the user and calculatinga distance of the user relative to the auxiliary positioning object;

capturing at least one image comprising the auxiliary positioningobject;

obtaining, according to the at least one image, direction information ofthe user relative to the auxiliary positioning object and positioninformation of the auxiliary positioning object; and

obtaining position information of the user according to the positioninformation of the auxiliary positioning object, the distance of theuser relative to the auxiliary positioning object, and the directioninformation.

For specific implementations of the units in the program 832, referencemay be made to the corresponding units in the embodiments shown in FIG.3 to FIG. 7, which are not described in detail herein. Persons skilledin the art may clearly know that, to make the description easy andconcise, specific working processes of the devices and modules are notdescribed in detail herein, and for details, reference may be made tothe corresponding process descriptions in the method embodiments.

The apparatus in the embodiments of the present application performsprecise positioning by acquiring a distance and a relative directionbetween a user and an auxiliary positioning object which the user isgazing at, to obtain a position of the user relative to the auxiliarypositioning object, thereby improving the precision of imagecapturing-based positioning.

Persons of ordinary skill in the art may aware that, the units andmethod steps of various examples described in the embodiments disclosedin this text may be implemented by using electronic hardware, or acombination of computer software and electronic hardware. Whether thefunctions are executed in a hardware or software form depends onspecific applications and design constraints of the technical solutions.For every specific application, persons skilled in the art may implementthe described specific function using different methods; however, theimplementation should not be considered as exceeding the scope of thepresent application.

If the function is implemented in a form of a software function unit andis sold or used as an independent product, the software function unitmay be stored in a computer readable storage medium. Based on thisunderstanding, the technical solution of the present applicationessentially, or parts contributive to the prior art, or parts of thetechnical solution may be embodied in a form of a software product. Thecomputer software product is stored in a storage medium comprisingmultiple instructions for enabling a computer device (such as a personalcomputer, a server, or a network device) to execute all or a part ofsteps of the method described in the embodiments of the presentapplication. The storage medium comprises: a USB flash disk, a mobilehard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), amagnetic disk, an optical disc, or another medium capable of storing theprogram code.

The implementations are only used for describing the presentapplication, instead of limiting the present application, and persons ofordinary skill in the art can make various changes and variationswithout departing from the spirit and scope of the present application,and therefore, all equivalent technical solutions fall within the scopeof the present application, and the protection scope of the presentapplication is defined by the claims.

1. An image capturing-based positioning method, comprising: determiningthat an eye of a user is gazing at an auxiliary positioning object;photographing the eye of the user and acquiring a distance of the userrelative to the auxiliary positioning object; capturing at least onefirst image comprising the auxiliary positioning object; obtaining,according to the at least one first image, direction information of theuser relative to the auxiliary positioning object and positioninformation of the auxiliary positioning object; and obtaining positioninformation of the user according to the position information of theauxiliary positioning object, the distance of the user relative to theauxiliary positioning object, and the direction information.
 2. Themethod according to claim 1, wherein the obtaining, according to the atleast one first image, direction information of the user relative to theauxiliary positioning object and position information of the auxiliarypositioning object comprises: sending the at least one first image to atleast one external device; and receiving, at least one of the directioninformation and the position information of the auxiliary positioningobject.
 3. The method according to claim 1, wherein the obtaining,according to the at least one first image, direction information of theuser relative to the auxiliary positioning object comprises: analyzingthe at least one first image, to obtain the direction information. 4.The method according to claim 3, the method further comprising:pre-storing at least one second image which is shot in at least onedetermined direction and includes the auxiliary positioning object,wherein the obtaining, according to the at least one first image,direction information comprises: analyzing the at least one first imageaccording to the at least one second image, to obtain the directioninformation.
 5. The method according to claim 1, wherein the obtaining,according to the at least one first image, position information of theuser relative to the auxiliary positioning object comprises: recognizingthe auxiliary positioning object in the at least one first image; andacquiring pre-stored position information of the auxiliary positioningobject.
 6. The method according to claim 1, wherein the photographingthe eye of the user and acquiring a distance of the user relative to theauxiliary positioning object further comprises: capturing at least onefundus image of the eye; adjusting at least one imaging parameter of anoptical path between a capturing position of the at least one fundusimage and the eye until an fundus image satisfying at least one setresolution criterion is captured; and analyzing the at least one fundusimage, to obtain imaging parameters of the optical path and opticalparameters of the eye corresponding to the fundus image, and acquiring adistance of a current gaze point of the user relative to the useraccording to the imaging parameters and the optical parameters of theeye.
 7. The method according to claim 6, wherein the adjusting at leastone imaging parameter of an optical path between a capturing position ofthe at least one fundus image and the eye until an fundus imagesatisfying at least one set resolution criterion is captured comprises:adjusting at least one of a focal length and a position of at least oneoptical device on the optical path.
 8. The method according to claim 6,wherein the adjusting at least one imaging parameter of an optical pathbetween a capturing position of the at least one fundus image and theeye until an fundus image satisfying at least one set resolutioncriterion is captured comprises: transmitting, corresponding to pupilpositions associated with different optical axis directions of the eye,an image presented at the fundus to the capturing position of the atleast one fundus image.
 9. The method according to claim 6, wherein thephotographing the eye of the user and acquiring a distance of the userrelative to the auxiliary positioning object further comprises:projecting a light spot pattern to the fundus.
 10. The method accordingto claim 1, the method further comprising: guiding the user to gaze atthe auxiliary positioning object.
 11. The method according to claim 10,wherein the guiding the user to gaze at the auxiliary positioning objectfurther comprises: marking the auxiliary positioning object.
 12. Themethod according to claim 11, wherein the guiding the user to gaze atthe auxiliary positioning object further comprises: marking theauxiliary positioning object by means of augmented reality.
 13. An imagecapturing-based positioning apparatus, comprising: a gaze determiningmodule, configured to determine that an eye of a user is gazing at anauxiliary positioning object; an eye photographing module, configured tophotograph the eye of the user and acquire a distance of the userrelative to the auxiliary positioning object; an image capturing module,configured to capture at least one first image comprising the auxiliarypositioning object; an information acquisition module, configured toobtain, according to the at least one first image, direction informationof the user relative to the auxiliary positioning object, and positioninformation of the auxiliary positioning object; and a positioningmodule, configured to obtain position information of the user accordingto the position information of the auxiliary positioning object, thedistance of the user relative to the auxiliary positioning object, andthe direction information.
 14. The apparatus according to claim 13,wherein the information acquisition module comprises a communicationunit, configured to: sending the at least one first image to at leastone external device; and receiving, at least one of the directioninformation and the position information of the auxiliary positioningobject.
 15. The apparatus according to claim 13, wherein the informationacquisition module comprises: an image analyzing unit, configured toanalyze the at least one first image, to obtain the directioninformation.
 16. The apparatus according to claim 15, wherein theinformation acquisition module further comprises: a storage unit,configured to pre-store at least one second image which is shot in atleast one determined direction and includes the auxiliary positioningobject, wherein the image analyzing unit is further configured toanalyze the at least one first image according to the at least onesecond image, to obtain the direction information.
 17. The apparatusaccording to claim 13, wherein the information acquisition modulecomprises: an object recognition unit, configured to recognize theauxiliary positioning object in the at least one first image; and anobject information acquisition unit, configured to acquire pre-storedposition information of the auxiliary positioning object.
 18. Theapparatus according to claim 13, wherein the eye photographing modulefurther comprises: a fundus image capturing sub-module, configured tocapture at least one fundus image of the eye; an adjustable imagingsub-module, configured to adjust at least one imaging parameter of anoptical path between the fundus image capturing sub-module and the eyeuntil an fundus image satisfying at least one set resolution criterionis captured; and an image processing sub-module, configured to analyzethe at least one fundus image, to obtain imaging parameters of theoptical path and optical parameters of the eye corresponding to thefundus image, and acquire a distance of a current gaze point of the userrelative to the user according to the imaging parameters and the opticalparameters of the eye.
 19. The apparatus according to claim 18, whereinthe adjustable imaging sub-module comprises: an adjustable lens device,having at least one of an adjustable image parameter and an adjustableposition on the optical path.
 20. The apparatus according to claim 18,wherein the adjustable imaging sub-module further comprises: acurved-surface beam splitting device, configured to transmit,corresponding to pupil positions associated with different optical axisdirections of the eye, an image presented at the fundus to the fundusimage capturing sub-module.
 21. The apparatus according to claim 18,wherein the eye photographing module further comprises: a projectionsub-module, configured to project a light spot pattern to the fundus.22. The apparatus according to claim 13, further comprising: an objectguiding module, configured to guide the user to gaze at the auxiliarypositioning object.
 23. The apparatus according to claim 22, wherein theobject guiding module further comprises: an object marking unit,configured to mark the auxiliary positioning object.
 24. The apparatusaccording to claim 23, wherein the object marking unit comprises: anaugmented reality marking sub-unit, configured to mark the auxiliarypositioning object by means of augmented reality.
 25. The apparatusaccording to claim 13, wherein the apparatus is a wearable device. 26.The apparatus according to claim 25, wherein the apparatus is a pair ofsmart glasses.
 27. A computer readable storage medium, comprisingexecutable instructions for: determining that an eye of a user is gazingat an auxiliary positioning object; acquiring an image corresponding tothe eye of the user and calculating a distance of the user relative tothe auxiliary positioning object; capturing at least one imagecomprising the auxiliary positioning object; obtaining, according to theat least one image, direction information of the user relative to theauxiliary positioning object and position information of the auxiliarypositioning object; and obtaining position information of the useraccording to the position information of the auxiliary positioningobject, the distance of the user relative to the auxiliary positioningobject, and the direction information.
 28. An image capturing-basedpositioning apparatus, comprising a central processing unit and amemory, wherein the memory stores computer-executable instructions, whenexecuted by the central processing unit, configured to perform thefollowing steps: determining that an eye of a user is gazing at anauxiliary positioning object; acquiring an image corresponding to theeye of the user and calculating a distance of the user relative to theauxiliary positioning object; capturing at least one image comprisingthe auxiliary positioning object; obtaining, according to the at leastone image, direction information of the user relative to the auxiliarypositioning object and position information of the auxiliary positioningobject; and obtaining position information of the user according to theposition information of the auxiliary positioning object, the distanceof the user relative to the auxiliary positioning object, and thedirection information.